Some notes before you get started...
This paper is long and technical and really not for the general user who just wants to understand what this "doubling" thing is. I strongly suggest that non-techies instead read the Doubling Theory section of my Wrye Mash documentation.
If you simply want to fix/prevent doubling, then download Wrye Mash and start using it. Wrye Mash is the only program that fixes/prevents doubling and similar problems. (This paper is actually the proposal paper for a program like Wrye Mash!)
This paper was written in December 2004, and based on my experience since then, there are a few inaccuracies and a few missing points. Some of that additional info available at UESP Wiki. Hopefully, I'll eventually have time to update this.
• ModIndex
• Notes
• Effects of Common Doubling Fixes
• Saving Outside Affected Cells
• Surgical Reference Correction
• Launcher
• Credits
• License
• Contact
• I was playing around with Enchanted Editor yesterday and that plus some more testing has given me some insights into how to avoid or fix doubling/deleting due to changes in mods and/or mod load lists. Some stuff can be done by hand, but a more general fix would require a program to fix game save files.
• BTW, I am not talking about deleting cell records, or unloading/reloading mods. Both of these things work (sort of, mostly) but a better program solution should be possible.
• *Warning: There may be some errors in the following document. TES is quite complicated and it's easy to take some new bit of information and from it come to expect something to be true – which turns out to not be true. In writing this, I've already corrected a couple of errors like this – but more may remain.*
• Note: I haven't had a chance to do a real edit of this. Sorry. RSN!
• Now, first, I am not talking about one mod replacing the database entry of another mod – which can also cause problems – but well known, fairly obvious problems. Here I am only talking about references...
• When you place an object in a cell, you're creating a reference. But objects in inventory, containers, leveled lists, etc. are not references.
• If you pull up the Data Files menu, and click on the details button, you'll see references listed under each cell "REFR".
• Note: The TEMP_REFS is not itself a reference, but a marker of the division betwen refs persist objects (above the lines) and non-persistent refs (below the line)
• Now, when you play the game, some items in cells get modified, and some don't. Creatures move around, get killed; doors open; items get taken. OTOH, statics just site there (unless some script specifically modifies them). If an item is changed, information about that change is saved to the game file. If it's not changed, then no info is saved to the game file. This info takes the form of a bundle of sub-records in the game save files. (They're "sub" records of the cell record.)
• One of these sub-records, the FRMR record is responsible for matching the game save (ess) record with the original esmp ("esmp" == "esm/esp") record. It does this through two numbers: the ObjectIndex and the ModIndex.
• If you have Enchanted Editor with the current template, then you'll see two fields: "ObjectIndex" and "flags." Flags would be better characterized as a "byte,1" called "ModIndex."
• If something happens that causes either of these numbers to get out of sync with the source esmp file, then you're likely to get doubling, deletion, weird behavior, or CTDs.
• Problems with ObjectIndex arise only from changing a mod and then using a "dirty save". So this section is only useful to modders – and to players who want to switch to a modified version of a mod.
• The ObjectIndex is the "number" of the reference. For esmp files that you create, the reference number starts at 1 for the first reference in your first cell and increments by one with each additional reference across all cells. If you pull up the detail list (from the Data Files dialog in TESCS), the first created reference on it is 1, second is 2, etc.
• But that's for created references only. Modified references (references that come from the master files that your esp depends on) keep the ObjectIndex numbers of the master files. This is (part of) how esps are able to modify references placed by esms.
• BTW, esps are not able to modify references placed by other esps. If you try to do so, you just get a completely new reference to the object. Thus when you load both esps, you'll see two objects instead of one. I suspect that this is because TESCS simply refuses to treat such changes as modifications. Instead it treats the change as a creation.
• I've heard that the same is true of esms – i.e., that esms cannot modify references placed by other esms. If so, then it's the same thing, TESCS simply treats the change as creation rather than a modification.
• Now, back to created object numbering... Unlike dialog id numbers, objectIndex numbers are re-generated every time you save your esp. So, if you add one more reference in your first cell, then the objectIndex numbers of all following references will be incremented by 1. If you have a savegame that refers to any of these objects whose ObjectIndex numbers have changed, then the engine will fail to match the ess references against the esmp references and you will see doubling, deletion, etc.!
• BUT, if you make changes that don't affect change the ObjectIndex numbers for already visited cells, then you will not have doubling problems – unless the ModIndex is changed – which it will be unless you use a tool to reset the modification date of the mod (see ModIndex below).
• In other words, there are a lot of changes that you can safely make.
• You can change any non-reference info. E.g., object data, dialog, etc. (Of course doing this might cause additional problems – but it won't cause doubling.)
• So, if the mod has no references at all, then no problem!
• You can safely add/change any reference that has not been modified in-game, and that you will not cause ObjectIndex renumbering in cells that you have already visited.
• Specifically, adding a completely new cell and stuffing it with references is safe.
• You can safely add/remove changes to any master file reference that has not itself been changed in game.
• You can move it – so long as you don't move it out of it's original cell.
• You can change its instance properties. E.g, ownership, lock level, etc.
• You can use "Search and Replace" to replace it with a completely different object. E.g., you can turn a rock into an NPC.
• You can delete it.
• However, you cannot cut and paste it. (I presume that this marks the original as deleted and then creates a new reference.)
• You can safely modify changed references so long as...
• You don't delete it, move it out of its cell or change its id.
• This applies to both created and master file references.
• However, your changes will likely not be reflected in the game, since the changes made in the game would override the changes to the esp file.
• Note: There may be some special cases where some types of in-game changes would be incompatible with your changes and cause CTD or other problems. For example, changing the number of variables in a script might easily cause odd behavior – and at least you'll get a warning about it.
• Some clarifications:
• Modified cells are placed into your esp file in the order that they exist in your referenced esm files. New cells that you create (even new exterior cells) are placed at the end of the cell list in the order that you create them.
• However... If, after adding your cells, you then add a dependency on another master file, then any changes to its cells will appear after your new cells.
• Of course, you can effectively "unvisit" a cell by deleting records of changes to it from your ess file.
• The standard trick is to unvisit all cells that your esp touches – but, in fact you only need to unvisit cells which would experience changes in objectIndex numbering. Of course, it may be simpler to just delete all visited cells.
• More Tricks
• Suppose that you're creating a mod and think that you might add an interior cell in Balmora later, but don't want to do so yet. Then you can create 2-3 invisible static objects (in the correct cell) in Balmora, and go ahead and play. Then when changing the mod later, you just use search and replace to replace these placeholder objects with the objects that you'll need (house plus door) and then create the interior cell that you need. Because you're replacing existing objects in Balmora, and because the new interior cell is added after all your existing cells, you don't mess up your existing ObjectInfo numbers.
• Programmatic Fix
• It should also be possible to write a program to help with mod merges
• Essentially the doubling problems with ObjectIndex are due to the fact that TESCS renumbers references every time you save the mod. However, it should be possible to write a program that allows you to merge two (or more) mods, while retaining the ObjectIndex numbering of the first mod. More on that below.
• The ModIndex is nothing more than the place that the mod occupied in the load-list at the time the game was saved.
• Note that this number is determined by the player, not the modder!
• Except when a modder releases a new version of their mod with a new date – this will cause a load order modification.)
• Unfortunately, this number is extremely fragile. Any changes to the load order (e.g., by adding or deleting mods or by changing the modified date of a an esmp file) will result in changes to the ModIndex assigned to at least some of esmps – and these will no longer match the corresponding ModIndex numbers in the ess file.
• While changes in ObjectIndex numbers cause ess changes to be applied to the wrong reference from the same esp, changes in ModIndex numbers cause ess changes to be applied to the wrong reference in different esps. E.g., if a ref placed by Alpha.esp is deleted in gameplay, a change to ModIndex could result in that deletion being applied to a ref placed by Beta.esp.
• ModIndex related doubling problems can essentially be prevented by avoiding changing load order. Which is pretty limiting, but some rules and tricks can help...
• Use a tool to manage the dates of your mods. This is particularly important if you modify a mod. Basically, once a mod date has been set, you want the tool to keep its modification date constant so that its load order does not change.
• If you add a new esmp to your load, it should come chronologically after esmps that you've already loaded. BTW, note that any new esms will certainly be added before any esps that you have loaded! I.e., even if an esm doesn't add new references, it can cause problems!!!
• If you remove an esmp, replace it with a dummy esmp which has the same date. This will cause esmps loaded after it to retain their existing ModIndex numbers.
• Note: Some quest mods play nice by providing "loot only" versions of themselves. The date of these loot only versions should be the same as the date of the full mod.
• You may want to pre-add several dummy esmps which can be replaced with real mods at a later date.
• E.g., you might have several quest slots. When you get a new quest, you stick it into one of theses slots. When you're done with the quest, then you replace the quest with another quest or replace the dummy esmp.
• Note that if the quest mod provides a loot mod (which should be database entries only), you don't want that mod eating the quest slot. So instead, you have a merged loot mod which you merge the quest's loot mod into.
• However this approach can still leave dirty elements – see note below.
• If you have more than one mod which you want to stuff into a slot, then you can combine those mods, and then stuff the combined mod in the slot.
• However You should not play with the mod, then merge other stuff into it and continue to play with it! This will cause ObjectIndex based doubling as described above.
• Unless it does not place any created references into the game world! Again see ObjectInfo above.
• E.g, you might setup your mod list like this:
Morrowind
Tribunal
Bloodmoon
Merged database only patch (no references)
...
Personal mods
Merged Loot mod
Quest Slot 1
Quest Slot 2
Quest Slot 3
...
• Now, when you get a new quest mod:
• Stuff it into one of the quest slots. (I.e., set its date to match the placeholder, then activate it in the place of the placeholder.)
• Play the quest mod. Then, when you're done.
• Extract it's loot and merge the loot into your merged loot mod. (Being sure to reset the date of the loot mod, of course.)
• Remove the quest mod, and replace it with the placeholder mod.
• Open a saved game and resave.
• However this approach can still leave dirty elements. If the quest mod places standard objects (objects from other mods) into standard cells of the gameworld, then changes to these objects (moves, deletions, etc.) will not be cleared by removing the quest mod. I.e., they could still mess up the next mod to occupy the slot.
• Things Modders Can Do...
• While ModIndex is determined by the player, there are things that Modders can do to reduce their likelihood of being affected by changes to it.
• Don't place references in cells that are likely to be used by other mods.
• For example Metal Queen Boutique is relatively safe because 1) it's way up north, and 2) it's actually not in the Dagon Fel cell, but rather just outside (if I remember correctly).
• If you need to place an item in a popular cell, use a script (e.g., a startup script) to place it there rather than TESCS. This will give it a ModIndex of 0, making it invulnerable to doubling, etc.
• Of course, this technique only works with items that have no reference specific info. Thus no load doors, locked chests, soul gems with pre-trapped souls, etc.
• ModIndex of ESPs
• So far, I've been talking about ModIndex in ess's. But how about ModIndexes in esps? These work essentially the same and have the potential to cause the same doubling problems as ModIndexes in ess's.
• For esps, the "load-list" is the master files that it depends on – in chronological order. If a change is made to a reference from a master file, then that change is given an FRMR with the ObjectIndex from the master file and a ModIndex that corresponds to the load position of that master file for that mod. E.g., if your mod depends on Morrowind and Bloodmoon, then references modified from Morrowind have ModIndex == 1 and references from Bloodmoon have ModIndex == 2.
• Notes:
• If (with the above example) you now add a dependency on Tribunal, you will get doubling – because now Bloodmoon is third on the esp's dependency list and so the ModIndex between the saved esp and the new dependency list is out of sync.
• If two esps modify the same master index reference, then doubling will not occur. Instead, the last mod to load wins. I.e., its changes override the changes from both the esm and the earlier loading esp.
• If you change a reference that comes from another esp then you'll get doubling – each ESP will contribute their own version of the reference. If you check mod file for your current esp, you'll see that the ModIndex for the reference modified from the other esp will be 0 – i.e., the current mod now thinks that it created that reference.
• If you examine a changed reference in-game, you'll see that the file that it's associated with is the esp, not the esm. And if you modify the reference, and save the game, you'll see that the ModIndex for your change corresponds to the esp, not the esm. So, apparently, there's a period while the esps are loading when a changed reference knows both which esp it's associated with and which esm it's changing. (Otherwise, the engine would not be able to resolve conflicts between two esps trying to modify the same esm reference.) But by the time the user sees it, the ModIndex associated with the changed reference is just the one due to the esp.
• So, rules to avoid problems:
• If you've changed a reference for a master file other than Morrowind (which always has ModIndex == 1), don't then add another dependency which will come earlier in the dependency list than your mod.
• Now, one other fly in the ointment... Save game headers...
• The file header info for each ess file (the TES3 record) includes sub-records which remember which esmp files the save game depends on as well as their size and load order.
• If any of these conflict with the info from the currently configured load order info, then:
• You'll get a warning message allowing to not load the save. Assuming that you go ahead and load, then...
• The ModIndex of ess references to esm placed references will be automatically corrected. (This is why esms don't double. See "ESM Files" below.)
• The ModIndex of ess references to esp placed references where the ModIndex now looks invalid (because the corresponding load number now has a differently named esp) will be reassigned.
• However, this reassignment seems to be unpredictable. Moreover, the reassignment may change if you save and reload the game. (And the time after that??)
• If you change the files size, but not the name or file position (i.e., if you modify the esp file, but somehow force it to keep the same load order), then the ModIndex apparently is not reassigned.
• Note that if you change the file name, but leave the file otherwise the same, the ModIndex will be reassigned.
• Once you save the game the headers will be synced with the current load order, but errors caused by ModIndex mis-assignments will remain.
• Using ORI
• If you bring up the console, click on an object, and type "ori" in the console, you'll get some reference info on the object. The PhysIdx is the ObjectIndex, and the file name is just the file that corresponds to the ModIndex.
• If the ModIndex is zero, then the reference "belongs" to the ess file, and no file is listed. Note that this is quite normal. Objects with no file listing (i.e., "owned" by the ess) include:
• Any object that has entered the game – or passed through – a container, inventory or barter.
• Note that when you pick up an item placed in the open by an esmp (i.e., not in a container), you don't actually get the object. Instead you're given a copy of the object and then the original object is deleted.
• Spawned creatures/NPCs.
• Any object placed by a script.
• Deleting References:
• If you delete a reference (e.g., use "disable", and then "set delete 1" in the console), then:
• If the object is owned by the ess, it will be deleted completely.
• If the object is associated with an esmp file, then it will remain, but be marked as deleted in the ess file at the next save.
• If you're using the console, the difference will be apparent. If it's an ess object, then the title of the console will immediately change back to "console", but if its an esmp-associated object, the title of console will continue to match the object – and in fact, you can undelete the object (though this is likely to cause additional problems – e.g., loss of position information.)
• ESPs vs. ESSs
• It's sometimes stated that ess's are just esps with a little extra information. While there's a fair amount of truth in this, its also true that each type of file is handled in ways that are substantially different from the other.
• Perhaps a better characterization would be that esps are halfway between esms and ess's.
• Save games remember changes to the game state. For objects placed in the open in the game world (references), the engine tries to match its changed references to the original references through the FRMR subrecord for that reference. If the match fails, then doubling, deletion, weirdness or CTD are all possibilities.
• In doing this matching, the game only considers the ObjectIndex and the ModIndex. So, if the save game remembers that a reference has been deleted, it will apply that deletion to the reference (in the cell) that has the matching ObjectIndex and ModIndex. If that match fails, then nothing will be deleted.
• This deletion may not be applied if the engine thinks that the ModIndex or ObjectIndex for that reference has changed. However, this "grace period" only applies to load immediately after a change in the load list or in the size of one of the files in the load list. If you save and reload, the deletion will be applied.
• Note: These deletion records do not go away! So even though they don't match any reference in the current configuration, they might match a reference in load configuration that you have months after they're created. I.e., you may not see a problem until months from when the causes of the problem were put in place.
• Note: In this demo, I'll assume at times that you have Enchanted Editor. If you don't have EE, you can get it, or just read along – you'll still see a lot of what I'm talking about even without EE.
• BTW, if you're looking for EE, there doesn't seem to be a link to an official site. Try searching Elder Scrolls forum or doing a google. In other words, apparently you have to go on a "quest" in order to get EE!
• In Wrye Demo (also available at Wrye Morrowind) you'll find (along with some other stuff) three doubling demo esps: Wrye Double 1, Wrye Double 2 and Wrye Double 3. Just ignore the other stuff and copy these three demos to your data files folder.
• Now, load up Wrye Demo 1 and 2 (but not Wrye Demo 2a ) files in your launcher and head to Seyda Neen. There, against the south facing wall of Vodunius Nuccius' house you'll find two red jars and two blue jars. The blue jars come from Wrye Double 1 and the red jars come from Wrye Demo 2.
• You can test this by bringing up the console, clicking on each of the jars in turn and using the ori command.
• When you do this, you might also note the PhysIdx numbers – these are the same as the ObjectIndex numbers discussed above.
• Now, take the upper red jar, and save your game to "Beta 1". (Or whatever you like, but I'll call it "Beta 1" from here on.)
• Now, drop the red jar and check its ori. Note that the file is now "None". This is because the reference that you just created by dropping a red jar from inventory is not the same as the jar you picked up. Rather it's a second generation copy (one copy into your inventory, and then one copy out of your inventory to form the reference now lying at your feet). If you were to save this game, unload Wrye Demo 1 and 2 and then reload this last save, the 3 remaining bottles against the wall would still be gone, but this reference would remain.
• Now, quit and open up enchanted editor.
• Before you open any files, go to Tools: File Format: Edit, and click through the warnings. (Or you may decided to chicken out and not mess with these settings – if you're a newbie, that may be wiser.) Go to the CELL.FRMR record, expand it and then look at the last record under it. If it says "flags","Bitfield,1", then change that to "ModIndex", "Byte,1". Then close the template definition. (You may have to save it first.)
• Now, open up "Beta 1.ess", go to the Cell Seyda Neen, -2, -9, and then go to the FRMR record for the red jar that you just picked up – I think that it will be the last set of sub-records under Seyda Neen, -2, -9. If you're totally clueless about EE, then now is the time to go through the docs a little bit.
• Ummm, by the way, the docs are distributed separately from the editor. Again check Elder Scrolls forum for info on where to find them if you don't have them.
• Now, check the FRMR for the reference. The object index should be 1, and the ModIndex should equal the load position of the Wrye Demo 2 esp.
• BTW, you might note that there aren't any references for the other three jars – that's because they haven't changed. If something doesn't change, then it doesn't go into the ess file.
• Back to the FRMR for the redware pot... Click on the ModIndex number and decrement the number by 1 – this should associate this deletion sub-record with Wrye Demo 1 instead of Wrye Demo 2.
• Now save Beta 1 in EE, launch Morrowind and reload it. Voila! Now the upper blue pot is missing instead of the upper red pot! BTW, note that the deletion takes place despite the fact the NAME_NEXT record says that it's deleting a redware pot, not a blueware pot. The engine doesn't care – it just looks at the ObjectIndex and the ModIndex.
• Now, quit, go back to EE, increment the ModIndex by one (back to it's original value) and save. Now, open up the launcher and unload Wrye Demo 2 and load Wrye Demo 2a in it's place. Now, launch and load your saved game. You'll get a warning that your files have changed. Ignore that and load.
• Now from this point on, I can't predict what will happen – I've had different things happen on subsequent tests. Basically, the ModIndex for the deletion seems to get reset to some new number.
• The new ObjectIndex/ModIndex pair may match some reference in the current cell – in which case that reference will disappear. Or it may not. I suspect that the ModIndex may end up fluctuating every time the engine thinks the load list has changed. I.e., if you save (to a new file) and reload at this point, some object might disappear which had not disappeared originally. If the new ModIndex happens to be the same as Wrye Demo 1 or 2a, then that should be obvious – you'll see stuff disappear in front of your eyes.
• Note: Do NOT save over Beta 1 – you'll need that the same for the next part of the demo)
• Now, quit out of Morrowind and go back to EE, and edit Beta 1.ess again. This time, go to the the File Header Record (there's just one: "TES3 Beta 1"), expand it and look at the MAST sub-record for "Wrye Double 2.esp". (Note that its position in the list corresponds to its load position.)
• Now, edit the MAST record to be "Wrye Double 2a.esp" (note: "2a", not "2"). Ordinarily, you would have to fix the DATA.FileSize property as well, but it happens that Double 2 and Double 2a have exactly the same file sizes.
• Now, save your changes to Beta 1, then launch Morrowind and reload Beta 1. This time, you get no load warnings, but the lower redware pot disappears! But this is actually the right thing to happen because in Wrye Double 2a, the order of the two pots is reversed, and hence their ObjectIndex numbers are reversed. (If you'll do an ORI on the top pot, you'll see that its PhysIdx is now 2 instead of 1)
• The point here is that by fixing the header list so that it looks right, the game engine thinks things are fine – and it doesn't randomly reassign the ModIndex number of deletions.
• ESM files seem to be immune to doubling, etc. Of course, since they're fixed, you would expect them to be immune from ObjectInfo caused problems. But it seems like ModIndex caused doubling should happen if a user generated esm slips in between Morrowind and Tribunal or Bloodmoon. But in fact, esm files seem to be immune to even this!
• There seem to be two reasons for this immunity:
• ObjectIndex numbers for references in standard esms are typically high (on the order of 100k). It's extremely unlikely that a mod is going to have an ObjectIndex that high, and even if it did, the chances of it's being in the same cell as the standard esm reference with that same number is extremely small. Hence a deletion from a missing/changed mod is extremely unlikely to be mistakenly applied to an official mod.
• When loading an ess for which the esm order mismatches the current esm order, the engine apparently fixes the ModIndex for all change records that affect esms. E.g., if you insert a new esm before Morrowind.esm, then Morrowind's ModIndex gets bumped from 1 to 2. The engine then, while loading the ess, changes every FRMR with ModIndex == 1 to have ModIndex == 2. And it act similarly for FRMR sub-records that refer to other master files. But it does not do this for FRMRs with ModIndexes that refer to esp files.
• Perhaps part of the reason for this is that the game cannot count on esps being present, and in this case more than a simple remapping of ModIndexes might be required. Still...
• One common piece of advice is that before changing your esp, you should save your game outside the cell being affected, but I haven't seen that this makes a difference.
• However, it may be that he 72 hour purge is having an effect here.
• The game engine purges some information if you have not encountered that object in the past 72 game hours. Notably, this happens for NPCs, who will forget that they talked to you before. Another example is the removal of the corpses of spawned creatures that you've killed. The purpose of this purging is to reduce the size of the ess file and the amount of information that the game has to process.
• So if it happens that you haven't visited a cell in 72 hours, then doubling may be less likely (because the doubling causing data has been purged).
• But this is just a guess – I haven't really tested it yet.
• If you simply remove an esp from your loadlist, load your game up and resave, then you'll purge all unique items, cells, etc. associated with that mod.
• Alternatively, if you have a loot mod for that esp, you can load up that loot mod in place of the original mod. This should allow you to keep the loot that you've collected while purging all other unique items.
• What happens here is that references for unique items are purged from all cell records in the ess. Not only are placement/movement records purged, but so are deletion records. I.e., the references are completely removed.
• Limitations
• Some references have matching records that are not part of the cell record. E.g., containers can have CNTC records associated with them. I don't know if these also get purged. But even if they don't, these garbage records will not cause doubling problems.
• Non-unique references are not removed. E.g., if the esp places a rampaging guar in the middle of Seyda Neen, that will not be removed during the purge/resave process. Moreover, deletion records of such non-unique items will continue to exist and may end up deleting references of another mod.
• I've heard that even some types of unique items will not be (fully) deleted. Apparently, if you see a headless NPC wandering around, this is where he comes from!
• The simplest and most effective approach to fixing doubling is to simply purge the ess of all cells – or at least of all cells that might be effected by mods that may have changed. This can be done with Enchanted Editor or with TESAME.
• The problem with this is that's it's pretty radical – it resets a lot of stuff and may well break some quests.
• Obviously, there's the danger of deleting all of the loot that you've collected. However, if all of your loot is sitting in a safe location (e.g., your house mod), then it's pretty safe to not delete the cell for that location.
• Obviously, this works because all change records (including deletions) are removed when you delete the cell records. No change records means no possibility of record mismatch!
• This method will remove headless NPCs!
• Using EE, you can go in and remove specific change deletion records from ess files. This allows very fine control (keep the good stuff, remove the bad) but obviously is pretty labor intensive.
• As the demo illustrated, it's possible to actually go in and fix FRMR records and ess headers so that the right things become deleted.
• Obviously this wouldn't be practical for a large number of references. Unless you had a program to do it automatically for you...
• Okay, finally we get the payoff... Given what we know above, it becomes clear that programs could be written to prevent and repair problems caused by ObjectIndex and ModIndex mismatching.
• The interface for this could be done several different ways, but an expanded launcher program with three tabs, whose functions are described below.
• First tab would be the Launcher...
• Interface
• Provides the same functionality as the Morrowind launcher, but also allows you to:
• View author, genre.
• Sort the launch list by load order, mod name, mod author, or mod genre.
• Reset the modification date to its last value. (There should be a checkbox that allows you to set this to happen automatically.)
• Change/set: esp name, date, author, genre.
• Co-start MW Enhanced, FPS manager, etc.
• Notes
• If you change the name of an esp, then you should have the option to automatically change the corresponding header entries in saved files. It should be possible to make this happen automatically. And, at the other extreme, it should be possible for the user to determine on a save by save basis whether the change is applied to that save.
• However, this may be too complicated. It may be simpler to defer this functionality to the save game manager tab (below).
• Second tab would the the Save Game Manager, which would have two functions: 1) Save Game Repair, and 2) LoadList Change Handling. The first function is concerned wit repairing ess files that have already been damaged by ObjectIndex and/or ModIndex changes. The second function is concerned with adapting a file to a changed loadlist before the game has been loaded under that loadlist.
• Interface
• List of save files, showing save name.
• Player name, health points, save time, save picture, might also be shown.
• Some of this info (e.g., the picture) might be shown in a sidebox or shown in a popup window for the currently selected or moused over game file.
• Flag graphics (check marks, or x's) should be shown indicating: 1) whether the save game has ever been been repaired, 2) whether the save game is compatible with the current loadlist.
• This list might allow sorting by any of the shown columns.
• Checkboxes to allow the user to select multiple files to operate on, and buttons to initiate either the Repair or Remapping function. However, multiple selection might not work well with the editing the ess's load list, which might be required. (See next.)
• The loadlist for the currently selected ess should be shown, and on it any esmps that can't be found should be marked. The user would have option of editing the name of any of the items in the loadlist. The edited name should appear in a different color (or be otherwise marked), and the "available" flag should reflect the availability of the new name. When the program already knows an old name to new name mapping (e.g., because the user changed it for another ess, or because the user originally changed it in the launcher) then it should automatically change the name or trivially allow the name to be changed.
• Note: If esp files have been merged (see Mod Merger below), then its possible for two or more different esps to be mapped to the same new name. This should be fine. Again if the Mod Merger is responsible for the merge, then the new name be applied automatically or semi-automatically.
• Approach: ModIndex Remapping
• There are several ways in which the loadlist can be changed which will require adaptation of the ess.
• The simplest of these changes is a change to the file size of an esmp, such as comes from modifying an esp, but somehow forcing it to save back to the same position in the loadlist. This change doesn't actually cause problems (so long as the changes to the esp don't cause problems – see suggestions above and Mod Merger tab below). But to avoid an error message during load of the game, then the file size recorded in the ess headers should be updated to the current file size.
• The next simplest change is a change of name. This will cause doubling and possibly other errors. But it can also be simply fixed by updating the appropriate header information in the ess file.
• On to the more complicated case... Real changes in load order...
• First, from the interface we now know the correct load order. We know that which esps have been added, removed and shifted around. From this we can deduce: which current ModIndexes should be deleted (esp removed) and which ModIndexes should be remapped to a new ModIndex, and for these, we know what the new ModIndex should be.
• So, from this its just a question of going through the esp and:
• Adjusting the headers to the reflect the new loadlist (with correct file names and sizes).
• For esps which have been reordered, remapping the ModIndex to the new value.
• For esps which have been deleted, deleting the FRMR and associated sub-records.
• For the latter case, there will sometimes be additional records present in the data file. Preferably, these will be removed as well. However it should not break anything if they are left in. Note that the current crude approach of deleting CELL records entirely fails to remove these auxiliary records – but this doesn't seem to cause too many problems.
• Approach: ESS Repair
• As mentioned previously, the game engine when applying ess files on load, checks the ObjectIndex and ModIndex, but does not check the reference id which is stored in the NAME_NEXT sub-record following the FRMR sub-record. In contrast, ESS Repair checks this record for validity and discards references that fail to match the available game state.
• First, the unplayed game state must be recreated.
• This is achieved by scanning through the cells and references supplied by esm and esp files as defined by the loadlist of the ess, in the order defined by the ess. (Note that this order may be different from the current loadlist and chronological ordering, but these problems can be fixed by using the ModIndex remapping function.
• The main complexity here is in applying references from esps that modify existing references from master files. The ModIndex from the esp file must be mapped to ModIndex in the ess loadlist in order to be applied correctly.
• And if a second esp then modifies the same reference, the reference created by the earlier esp must be removed and the new reference put in its place.
• At the end, there should be a list of references for each cell with each reference having ObjectIndex, source ModIndex and reference id.
• Note: If the user is repairing multiple data files, this list can be re-used for all other data files that have the same dependency list.
• Second, the ess's references are scanned through and compared against this list. If the id specified by the ess fails to match the id in the list, then that reference in the ess should be deleted. As for ModIndex remapping it would be desirable to also delete any associated sub-records outside of the CELL records, but this does not seem to be critical.
• The last tab (or perhaps it should be first) would be a mod merger.
• Background
• In merging mods (esps), there are (at least) three problematic areas: the headers, dialog/journal and references. I deal here with merging headers (and associated ModIndex numbers) and references, but I won't address dialog, which has its own problems. And there may be other types of references and/or coordination that I'm not aware of.
• As discussed above, the major source of doubling problems that arisen from modifying an already-in-use esp is the reference renumbering that TESCS does when it saves the esp. If you could merge esps without modifying their reference numbers, then you would not have a doubling problem.
• OTOH, you might now have references in the same cell with the same ObjectIndex number – which would cause problems. Thus at least these references would have to be renumbered.
• But if in doing this, you preserve the references numbers of the first esp being merged, then users who already have dependencies on that esp, but not on the other esps will not experience doubling, deletion, etc.
• Note that this means that released mods can be upgraded without breaking save games.
• Note that if the user has dependencies on more than the first mod being merged, then the Save Game Manager will need to be used to remap the ModIndexes to the new values. E.g., if the user is using Mod X plus additions Mod X.1 and Mod X.2. Then after merging, each of these dependencies would need to be remapped to the same merged mod (Mod X.All).
• Interface
• A list of mods, similar to that in the launcher tab, would be provided.
• User would then select the mods to be merged (e.g., using a checkbox column) and hit a "Merge" button.
• A dialog would ask for the name of the resulting merge file. Or the user might select that the changes be merged into the first esp, thus keeping its header information and replacing it in the load list.
• Esps would be merged in chronological order, and the merged esp would be set to have a modification date equal to the modification date of the first (base) esp being merged.
• Approach
• If the mods have different master file dependencies, then these dependencies would have to be merged.
• The new header file list would be created and the corresponding ModIndex numbers would be determined.
• Then the references with ModIndex != 0 in all esps would have to be remapped to have the corrected ModIndex.
• E.g., suppose esp 1 depends on Morrowind + Tribunal, while esp 2 depends on Morrowind + Bloodmoon. Then for esp 2, it would be necessary to remap ModIndex == 2 to ModIndex == 3. (Esp 1 would not have to be corrected because its existing mapping is correct.)
• Note that sometimes multiple mods change the same master file reference. When merging such changes, only the change made by the last merged mod should be kept.
• To merge the mod's regular references..
• Start out with the base esp Scan through and record the highest ObjectIndex number in it (usually the last reference in the last cell). Call this number "LastIndex".
• Now merge in the references from the next esp. If there is there is no conflict between the ObjectIndex in this esp and any ObjectIndex provided by the base esp in the same cell, then keep the new esp's ObjectIndex number. If there is a conflict, then increment LastIndex and set the ObjectIndex of the new esp's reference with LastIndex.
• This approach will keep all of the ObjectIndex numbers of the base esp, and most of the ObjectIndex numbers of the new esp in the merged esp.
• Thus a save game that depends only on the base esp will work fine with the merged esp. This would allow mod developers to release updated versions of their esps that would not break existing save games.
• However, if the save game depends also depends on the second esp, then it may experience some problems. (Though these problems can be partially ameliorated by using the repair function of the save game manager.)
• If there are additional esps to merge, then just repeat the process with them.
• Note: This approach guarantees unique ObjectIndex numbers within each cell, but not within the game. AFAIK, this is okay. But maybe it's not – maybe the engine requires unique objectIndex numbers within each esmp. If that's the case, then all of the second esps new references would have to be changed, and people with dependencies on it would have more problems (again these could be strongly ameliorated by using the the repair function of the save gam manager).
• Note: Even this more conservative approach still leaves ObjectIndex numbers out of sequence. AFAIK, this is okay – standard esm files have out-of-sequence ObjectIndex numbers. However, maybe esp files are treated differently, and its not allowed for them. In which case, this whole approach would not work.
• Additional Merging
• Of course in merging, all other records have be merged as well. Dialog in particular can be problematic. This problem is left as an exercise for he students.
• (C) 2004, by Wrye
• Feel free to mirror this.
• Home Page: http://wryemusings.com/
• For discussion of Wrye mods, etc., visit Elder Scrolls forums. My id there is "Wrye".
• However, like most Morrowind modders, I already spend waaaaayyyy too much time modding. So, I may not be very responsive.